This invention relates to a novel architecture for a network interface device for a data processing system and a method for exchanging network data between the processors of a data processing system, and in particular a blade server. A novel architecture for a PCIe device is also disclosed.
A conventional data processing system comprising one or more processors and a memory is attached to a network by means of a peripheral network interface device. The network interface device allows the data processing system to receive and transmit data over the network. The data processing system and network interface device communicate over a peripheral data bus, such as PCIe.
An Ethernet network interface device includes a controller, which typically forms data packets for transmission over the network, controls the data flows passing through the device and coordinates the transfer of data over the data bus between the data processing system and the network interface device. The physical transfer of data over the network is handled by a so-called “PHY” of the network interface device, which performs the signalling of network data over the network at the physical level of the Ethernet specification.
In a typical data processing system, several data buses may be involved in the transmission or reception of data over the network. Often there are different buses between the processor and memory, local storage devices and peripherals such as the network interface device. Data transfer over each bus has an associated overhead, and, along with the PHY, the various data bus circuits are typically amongst the most expensive components in terms of introduced latency and power requirements during data transmission or reception over a network.
Furthermore, the rate of transmission of data packets over the fabric of the network itself is limited by the network bandwidth, which imposes a maximum data rate between hosts on the network. This can be particularly limiting if the network fabric is used to connect the blades of a blade server.
Conventional network interface devices are provided as peripherals which are connected to a data processing system over a peripheral bus, such as PCIe. The network interface device may be a standalone peripheral card or it may be supported at the motherboard of the data processing system, but the device is accessed by the data processing system over the appropriate bus. This configuration is shown in FIG. 1, in which the CPU 103 of the data processing system 101 accesses the network interface device (NIC) 102 over a PCIe bus 106. The interface between the CPU and the PCIe bus is provided by chipset 105, which also typically provides direct memory access (DMA) to the memory by means of a memory controller. Network interface device 102 comprises a controller 107 and a PHY 108—the controller largely handling data flows at the link layer of the network protocol in use over network 109, and the PHY handling the transmission and reception of network data packets over the network in accordance with the physical layer requirements of the network protocol.
Since the receive and transmit queues for applications executing at processor 103 are located at memory 104, the network interface device must perform reads and writes directly to the memory in order to transmit and receive data over the network. Typically two or three memory accesses are required to effect each transmit operation, and one or two memory accesses are required to effect each receive operation. Often a network interface device will be operable to read and write to the memory by means of direct memory access (DMA), but all such accesses must still be performed over the peripheral data bus. The power consumption of each transmit or receive operation is not insignificant because each access must traverse the serialisation/deserialisation (SERDES) circuitry at the chipset 105 (which provides the interface to the bus for the processor/memory controller) and the SERDES circuitry at the network interface device (which provides the interface to the bus for the NIC). For a high speed network interface device, thousands of transmit and/or receive operations can be performed every second and the total power consumption of such an architecture can become very large. Similarly, the repeated memory accesses over the peripheral data bus introduces a significant latency to the transmit and receive operations performed by the network interface device.
For certain applications, System-on-a-Chip (SoC) devices have been developed which include a processing core and a network interface device, thus placing the network interface device closer to a memory accessible to the processor. For example, the STMicroelectronics STM32W family of SoC devices provides a 32-bit ARM processor and an integrated IEEE 802.15.4 radio. However, integrating a full network interface including a PHY into a processor substantially increases the power required by the processor and hence the amount of heat which must be dealt with at the processor. Furthermore, the signals handled at the PHY introduce a significant amount of additional electrical interference into the processor, making an SoC architecture unsuitable for a high speed CPU.
Improved architectures for allowing the components of a data processing system (such as a CPU and network interface device) to communicate with one another have been proposed, but none provide a low power architecture for a network interface having a low latency data path to host memory. Previous attempts to improve the architecture for internal communications within a data processing system include the IMS T800 Transputer presented in “IMS T800 Architecture”, INMOS Technical Note 6, INMOS Ltd, January 1988, which is published at http://www.transputer.net/tn/06/tn06.html. Other such architectures are described in Finn, Gregory G. “An Integration of Network Communication with Workstation Architecture”, ACM Computer Communication Review, October 1991, and Hayter, M., McAuley, D., “The Desk Area Network”, ACM Transactions on Operating Systems, pp. 14-21, October 1991.
Previously proposed schemes describing Ethernet protocol encapsulation are described in U.S. patent application Ser. Nos. 10/548,121 and 12/105,412.
There is therefore a need for an improved network interface device architecture having lower power requirements and a lower latency data path to host memory.